Assisting execution of manual protocols at production equipment

ABSTRACT

One variation of a method for assisting execution of manual protocols at production equipment includes: identifying a site occupied by a mobile device based on a geospatial location of a device; identifying a space within the building occupied by the device based on identifiers of a set of wireless access points wirelessly accessible to the device and known locations of wireless access points within the building; loading a protocol associated with an equipment unit in the space; calculating a position of the device within the space based on positions optical features, detected in a field of view of an optical sensor at the device, relative to reference features represented in a space model of the space; and, when the position of the device falls within a threshold distance of a reference location proximal the equipment unit defined in a step of the procedure, rendering guidance for the step.

CROSS-REFERENCE TO RELATED APPLICATIONS

This Application is a continuation application of U.S. Non-Provisionalapplication Ser. No. 16/425,782, filed on 29 May 2019, which claims thebenefit of U.S. Provisional Application No. 62/677,562, filed on 29 May2018, which is incorporated in its entirety by this reference.

TECHNICAL FIELD

This invention relates generally to the field of equipment operation inmanufacturing and more specifically to a new and useful method forassisting execution of manual protocols at production equipment in thefield of equipment operation in manufacturing.

BRIEF DESCRIPTION OF THE FIGURES

FIG. 1 is a flowchart representation of a method;

FIG. 2 is a flowchart representation of one variation of the method;

FIG. 3 is a schematic representation of one variation of the method; and

FIG. 4 is a graphical representation of one variation of the method; and

FIG. 5 is a flowchart representation of one variation of the method.

DESCRIPTION OF THE EMBODIMENTS

The following description of embodiments of the invention is notintended to limit the invention to these embodiments but rather toenable a person skilled in the art to make and use this invention.Variations, configurations, implementations, example implementations,and examples described herein are optional and are not exclusive to thevariations, configurations, implementations, example implementations,and examples they describe. The invention described herein can includeany and all permutations of these variations, configurations,implementations, example implementations, and examples.

1. METHOD

As shown in FIG. 1 , method S100, for assisting execution of manualprotocols at production equipment includes: accessing a geospatiallocation of a mobile device in Block S110; identifying a site containingthe geospatial location of the mobile device based on the geospatiallocation of the mobile device in Block S112; accessing wirelessconnectivity of the mobile device to nearby wireless access points inBlock S120; estimating a particular location within a particularbuilding occupied by the mobile device based on wireless connectivity ofthe mobile device to nearby wireless access points and known locationsof wireless access points within the particular building in Block S122;loading a space model representing optical features of the particularroom onto the mobile device in Block S130; at the mobile device,confirming presence of the mobile device in the particular room based onalignment between the space model and optical features detected in afield of view of an optical sensor integrated into the mobile device inBlock S134; in response to confirming presence of the mobile device inthe particular room, loading a protocol associated with an equipmentunit in the room onto the mobile device in Block S140; at the mobiledevice, tracking a position of the mobile device relative to theequipment unit based on the space model and optical features detected inthe field of view of the optical sensor in Block S142 and rendering aguidance for a step in the protocol, aligned with a correspondingelement of the equipment unit, on a display coupled to the mobile devicein Block S144.

One variation of the method shown in FIG. 5 includes: accessing ageospatial location of a mobile device in Block S110; identifying aparticular site occupied by the mobile device based on the geospatiallocation in Block S112; accessing identifiers of a first set of (i.e.,one or more) wireless access points (hereinafter “APs”) wirelesslyaccessible to the mobile device in Block S120; based on identifiers ofthe set of wireless APs and known locations of wireless APs within theparticular building, identifying a particular space within theparticular building within the particular site occupied by the mobiledevice in Block S122; loading a protocol associated with an equipmentunit in the particular space in Block S140, the protocol defining asequence of steps for operation of the equipment unit by a user;detecting a first set of optical features in a field of view of anoptical sensor integrated into the mobile device in Block S132;calculating a first position of the mobile device within the particularspace based on positions of the first set of optical features relativeto a constellation of reference features represented in a space model ofthe particular space in Block S134; and, in response to the firstposition of the mobile device falling within a first threshold distanceof a first reference location proximal the equipment unit associatedwith a first step of the protocol, serving a first guidance for thefirst step to the user in Block S144

Another variation of the method shown in FIG. 5 includes: accessing ageospatial location of a mobile device in Block S110; identifying aparticular site occupied by the mobile device based on the geospatiallocation in Block S112; accessing identifiers of a first set of wirelessAPs wirelessly accessible to the mobile device in Block S120; based onidentifiers of the set of wireless APs and known locations of wirelessAPs within the particular building, identifying a particular spacewithin the particular building within the particular site occupied bythe mobile device in Block S122; detecting a first set of opticalfeatures in a field of view of an optical sensor integrated into themobile device in Block S132; confirming presence of the mobile device inthe particular space based on alignment between the first set of opticalfeatures and a constellation of reference features represented in aspace model of the particular space in Block S134; following confirmingpresence of the mobile device in the particular space, loading aprotocol associated with an equipment unit in the particular space inBlock S140, the protocol defining a sequence of steps for operation ofthe equipment unit by a user; tracking a position of the mobile devicerelative to the equipment unit based on locations of the first set ofoptical features detected in the field of view of the optical sensor andthe constellation of reference features represented in the space modelin Block S142; and, at the mobile device, in response to the position ofthe mobile device falling within a first threshold distance of alocation proximal the equipment unit associated with a first step of theprotocol, serving a guidance for the first step to the user in BlockS144.

Yet another variation of the method shown in FIG. 1 includes: accessinga geospatial location of a mobile device in Block S110; identifying aparticular site occupied by the mobile device based on the geospatiallocation in Block S112; accessing identifiers of a first set of wirelessAPs wirelessly accessible to the mobile device in Block S120; based onidentifiers of the set of wireless APs and known locations of wirelessAPs within the particular building, identifying a particular spacewithin the particular building within the particular site occupied bythe mobile device in Block S122; in response to identifying theparticular space occupied by the mobile device, loading a space modelrepresenting a constellation of reference features within the particularspace; loading a protocol associated with an equipment unit in theparticular space in Block S140, the protocol defining a sequence ofsteps for operation of the equipment unit by a user; detecting a firstset of optical features in a field of view of an optical sensorintegrated into the mobile device in Block S132; calculating a positionof the mobile device relative to the equipment unit based on positionsof the first set of optical features relative to the constellation ofreference features represented in the space model in Block S134; and atthe mobile device, initiating a first step, in the protocol, associatedwith a particular element on the equipment unit and rendering a guidancefor the first step, visually aligned with the particular element on theequipment unit, on a display coupled to the mobile device based on theposition of the mobile device in Block S144.

2. APPLICATIONS

Generally, the method S100 is described below as executed by a mobiledevice and a computer system to serve protocols for testing andoperating biotechnical and/or pharmaceutical production workspaces wherethe operator receives guidance and instructions for the performance of aplurality of guided work instructions, tasks, and/or protocols relatedto the manufacturing of a product, the assembly of a product, performinga checklist, and other step-by-step procedures. In this embodiment theoperator is using equipment (hereinafter “equipment units”), which maynecessarily be disconnected from local area networks and local wirelessnetworks in order to limit opportunity for remote hacking orexploitation of these equipment and which may therefore require manualinput and manual recordation of data during testing and operation. Morespecifically, the mobile device and the remote computer system cancooperate to provide real-time guidance to an operator performing a testor operational protocol at an equipment unit by automatically: trackingthe position of the mobile device in space relative to the equipmentunit; presenting a sequence of prompts related to the protocol within anaugmented reality environment; highlighting—within the augmented realityenvironment—manual input regions and data capture regions on theequipment unit according to steps of the protocol; recording video ofselect steps within the protocol; reading quantitative and/orqualitative values from the equipment unit according to a current stepof the protocol; and/or recording these values, such as to an electroniclab notebook. The mobile device can thus execute Blocks of the methodS100 in order to form a one-way physically-isolated gateway forautomatically collecting data from equipment units and can define anoperator portal through which real-time, spatial guidance is served toan operator as the operator manually interfaces with these equipmentunits, thereby: streamlining testing and operation of these equipmentunits; reducing opportunity data capture and manual input error; andimproving operator efficiency without substantively reducing security ofthese equipment units.

In order to minimize latency, achieve high spatial accuracy, and thusserve accurate spatial guidance to an operator and collect targeted datafrom an equipment unit, the mobile device may cooperate with the remotecomputer system to execute Blocks of the method S100. More specifically,the remote computer system can execute Blocks of the method S100: toisolate the mobile device to a particular site based on a geospatiallocation (e.g., a “GPS” location) of the mobile device; to isolate themobile device to a particular floor or subarea of a particular floor ofa particular building on the site based on wireless APs that are withinwireless range of the mobile device; to isolate a particular room inthis particular building likely to be occupied by the mobile devicebased the mobile device's wireless connectivity to these wireless APs;and to then selectively upload a 2D or 3D model (e.g., a “localizationmap”) of this particular room to the mobile device. The mobile devicecan then: collect optical data (e.g., 2D color images, depth images, 3DLIDAR scans, infrared images, multispectral images) of the space itoccupies; extract features from these optical data; and then calculateits pose (i.e., its position and orientation) within the room—such asrelative to an equipment unit in the room—by locating these featureswithin the localization map of the particular room.

For example, the remote computer system can: localize the mobile deviceto a site, a building, a floor, and/or to a subarea of a floor of abuilding based on geospatial and wireless connectivity data collected bythe mobile device; predict a particular room in the building occupied bythe mobile device according to these wireless connectivity data; andthen selectively push a localization map of the room to the mobiledevice in order to enable the mobile device to localize itself to a highlevel of position accuracy and with low latency. Upon receipt of alocalization map of a room (likely to be) occupied by the mobile device,the mobile device can track its pose (i.e., location and orientation)within the room by regularly sampling its integrated or connectedoptical sensor, extracting features from images recorded by the opticalsensor, and calculating transforms that project these features ontocorresponding reference features in the localization map. Therefore,rather than store a localization map for an entire site, building, orfloor of a building over an extended period of time and downloadingupdates to this localization map over time, the mobile device candownload a localization map for a particular room (or other subregion)of a floor of a building on this site when geospatial and wirelessconnectivity data indicate that the mobile device is present in thisparticular room, thereby minimizing memory allocation for a localizationmap on the mobile device. The mobile device can then locally track itspose within the room based on the localization map and features detectedin the field of view of its optical sensor, thereby minimizinglocalization latency and data uploaded from the mobile device. Themobile device can then discard this localization map when the mobiledevice exits the room or at the end of a current shift, etc. When themobile device returns to this room at a later time, the mobile devicecan cooperate with the remote computer system to re-download alocalization map for the room, such as an updated localization mapregenerated based on new optical data collected from computing devicespassing through the room since the mobile device last downloaded alocalization map of the room.

Furthermore, once the mobile device calculates its pose within theparticular room and thus confirms its presence in this room, the mobiledevice can: automatically retrieve a protocol for an equipment test orproduction process on a particular equipment unit adjacent the mobiledevice; and render augmented reality content—aligned to featuresdetected in the field of view of the headset—on the mobile device, aconnected augmented reality device, thereby providing real-time,localized guidance for completing this protocol to an operator. Forexample, the mobile device can insert a prompt, icon, or bounding boxspatially-aligned to a material input, valve, dial, lever, keypad, orother input region on the equipment unit—specified in a current step ofthe protocol—in an augmented reality environment rendered on a heads-updisplay connected to or integrated into the mobile device, therebydirecting the operator wearing the heads-up display to this inputregion. The mobile device can also insert a prompt, icon, or boundingbox spatially-aligned to a material window, dial, or other display onthe equipment unit—specified in a current step of the protocol—in theaugmented reality environment rendered on the heads-up display, therebydirecting the operator to manually record a value from the display or tomove the mobile device near enough to the display to automatically readand record a value from the display.

3. SYSTEM

Generally, Blocks of the method S100 can be executed by a systemincluding: a computer system, such as a remote server or a computernetwork; and a local computing device, or connected to an augmentedreality headset. For example, the mobile device can include a heads-updisplay, eyes-up display, head-mounted display, or smart glassesconfigured to render augmented reality content for an operator wearingthis mobile device. Alternatively, the mobile device can include aWi-Fi-enabled smartphone or tablet connected to a separate augmentedreality device.

Furthermore, the mobile device can include: a suite of sensors (e.g., anaccelerometer, gyroscope, compass, gravity sensor, barometer, proximitysensor, magnetometer, ambient light sensor) configured to collectinformation about the mobile device's environment; local memoryconfigured to temporarily store a localization map of a room; and acontroller configured to execute Blocks of the method S100, includingdetermining a location of the mobile device in real space based on thelocalization map and data collected by the suite of sensors. Forexample, the mobile device can include: a depth camera paired with a 2Dcolor camera; or a pair of stereoscopic 2D color cameras. Each of theseoptical sensors can output a video feed containing a sequence of digitalphotographic images (or “frames”), such as at a rate of 20 Hz, and thecontroller can compile concurrent frames output by these optical sensorsinto a 3D point cloud or other representation of surfaces or features inthe field of view of the mobile device. Following receipt of alocalization map of a room occupied by the mobile device and generationof a 3D point cloud (or other representation of surfaces or features inthe field of view of the mobile device), the controller can implementpoint-to-plane fitting or other techniques to calculate a transform thatmaps the 3D point cloud onto the localization map in order to determinethe pose of the mobile device within the room.

However, the mobile device can include any other type of sensor in anyother quantity and can implement any other method or technique tocalculate its pose within a room based on a localization map of the roomand data recorded by these sensors.

4. SITE MAP AND INTERIOR MODEL

The remote computer system can generate and maintain a map of a site,including geospatial, wireless, and optical features representing thesite, buildings on the site, floors within these buildings, and roomswithin the floors at different levels of resolution. For example, acampus can be defined by a geospatially-referenced site boundary orassociated with a geospatial site reference point within a site map;each building on the campus can be defined by a geospatially-referencedbuilding boundary, associated with a geospatial building referencepoint, and/or associated with a set of (i.e., one or more) uniqueidentifiers of wireless APs (e.g., Wi-Fi routers) located in thisbuilding; each floor in this building can be associated with a set ofunique identifiers of wireless APs located on this floor; and each roomon this floor of the building can be associated with a unique identifierof particular wireless APs located in this room or associated withunique identifiers of wireless APs within wireless range of this room(i.e., detectable by wireless-enabled devices occupying this room) andlinked to a model (e.g., a “localization map”) and specific to thisroom.

The remote computer system can thus interface with an administrator—suchas through an administrator portal accessed through a native applicationor web browser executing on the administrator's computing device—togenerate and maintain a hierarchy of data representing this site,including: a geospatial map of the site and building on the site;wireless site surveys of other wireless access maps of floors and roomsinside these buildings; and discrete space models for individual rooms,clusters of rooms, or sub-volumes of rooms on these floors.

4.1 Layered Site Map

In one implementation shown in FIG. 2 , a site map includes a set oflayers representing a site, buildings within the site, floors withinthese buildings, and rooms within these floors. In this implementation,to construct the site map, the remote computer system can: retrieve avirtual map of a geographic region containing the site; extract a mapsection encompassing the site in the virtual map, such as automaticallybased on a site address supplied by the administrator or based on aboundary of the site drawn manually over the map by the administratorvia the administrator portal; extract geospatial reference markers forthis map section from the virtual map; and store the georeferenced mapsection in a site layer of the site map.

4.1.1 Building Layers

The remote computer system can extract boundaries around discretebuildings from the isolated map section, such as by implementingcomputer vision techniques to detect perimeters of these buildings inthe map section or based on perimeters of these buildings drawn manuallyover the map by the administrator via the administrator portal. Theremote computer system can then: extract geospatial reference markersfor each of these buildings; generate a building layer for selectbuildings on the site; populate each of these building layers with ageoreferenced representation of its corresponding building; and loadthese building layers into the site map. (Alternatively, the remotecomputer system can annotate the site layer with georeferencedrepresentations of each of these buildings.) The remote computer systemcan additionally or alternatively: access an architectural orengineering plan of the site; and extract a site layer and/or a set ofbuilding layers for buildings on the site from this architectural orengineering plan of the site.

4.1.2 Floor Layers

For each of these buildings, the remote computer system can then accessan architectural plan, an engineering plan, a utilities plan, and/or awireless site survey for the building, such as uploaded by theadministrator via the administrator portal or selected from a remotedatabase by the administrator via the administrator portal. The remotecomputer system can: extract a floor plan of select floors in thebuilding from the architectural or engineering plan, such asautomatically or with supervision of the administrator; generate a floorlayer for each of these floors in the building; annotate each floorlayer with unique identifiers and locations of wireless APs indicated inthe wireless site survey for the building; and load these buildinglayers into the site map. For example, the remote computer system can:intake the wireless site survey in the form of a table or spreadsheetuploaded or selected by the administrator via the administrator portal;extract locations and unique identifiers of wireless APs throughout afloor, a building, or the site generally from this table or spreadsheet;and populate corresponding floor layers in the site map with locationsand unique identifiers of wireless APs present on these floors.(Alternatively, the remote computer system can: access a wireless sitesurvey containing a map or gradient of wireless signal strength, uploadspeed, download speed, and/or wireless signal quality, etc. throughout afloor, throughout a building, or throughout the site; and load aparticular segment of a wireless site survey corresponding to aparticular floor into a floor layer for this particular floor.)

4.1.3 Room Layers and Space Models

Furthermore, the remote computer system can access 2D or 3Drepresentations of optical features of select rooms on select floors inselect buildings on the site, such as uploaded by the administrator viathe administrator portal or selected from a remote database via theadministrator portal. For example, the remote computer system can:interface with the administrator through the administrator portal toidentify rooms of interest (e.g., rooms in which equipment linked tovirtual-assist protocols is present); and access a database of opticalfiducials (e.g., passive QR codes or barcodes, active light elements)present in these rooms of interest, such as: immutable optical fiducialsinstalled in these rooms and their 2D or 3D locations; optical fiducialspresent on fixed equipment located in these rooms and their 2D or 3Dlocations; and/or optical fiducials applied to mobile equipmentfrequently assigned to these rooms or available in these floors orbuildings. The remote computer system can then: initialize a room layerfor a room in this set; and insert representation of optical fiducialsrepresenting this room, optical fiducials present on equipmentpermanently located in this room, and/or optical fiducials present onequipment commonly stored in this room or available on the same floor orbuilding into the room layer. The remote computer system can thusgenerate a room layer (or “space model”) that defines a low-resolutionrepresentation of the room, including a 2D or 3D map of immutableoptical fiducials in the room and a table of optical fiducials that maybe present in the room at any given time.

The remote computer system can additionally or alternatively access a 2Dor 3D optical scan of a room and then load this 2D or 3D optical scan ofthe room into the corresponding room layer. For example, this 3D opticalscan of the room: can include a 3D point cloud output from a depthsensor or generated by combining stereoscopic 2D color images recordedby 2D color cameras previously articulated within the room; canrepresent fixed walls, fixed or immobile equipment (desks, manufacturingequipment, lab equipment) present in the room, and doorways within theroom; and can include a table of pointers to mobile equipment that maybe present in the room, floor, or building, such as chairs, movingcarts, or material containers. The remote computer system can load this“high-resolution” 3D optical scan into the room layer for thecorresponding room, and this room layer can thus define or include a“space model” representing optical features that are or may be presentin the corresponding room at any given time.

For example, a space model can include a 3D localization map thatrepresents immutable surfaces within a corresponding room of interest,such as including floor, wall, and ceiling surfaces and surfaces ofequipment units and other fixed equipment located in the particularroom. In this example, the space model can also include 2D models, 3Dmodels, template images, or other representations of mobile equipmentand other objects: commonly present in the corresponding room, on thecorresponding floor, in the corresponding building, on the correspondingsite; specified in protocols for equipment units in the correspondingroom; or specified in protocols currently pending (i.e., currentlyincomplete) for equipment units in the corresponding room. The remotecomputer system can incorporate this space model into a room layer forthe corresponding room.

Additionally or alternatively, a space model for a particular space caninclude a map of wireless signal strengths of local, short-rangewireless transmitters (e.g., RFID tags, RFID interrogation antennas,wireless beacons) located on or near equipment units in the particularspace. This in this implementation, the mobile device can: identify thesite it occupies based on its coarse geospatial location; identify afloor building or building generally occupied by the mobile device basedon longer-range wireless signals received from wireless access points inthe building; track signal strengths of local, short-range wirelesstransmitters; and then calculate a fine (e.g., high-resolution, highprecision) position of the mobile device within the particular spacebased on these wireless signal strengths. For example, the mobile devicecan implement triangulation techniques to transform wireless signalstrengths of wireless signals received from these short-range wirelesstransmitters and known locations of these wireless transmitters in thebuilding in a position of the mobile device in the particular space.(Similarly, the mobile device can calculate its position relative to aparticular equipment unit or relative to a particular wirelesstransmitter in this constellation of short-range wireless transmitters.)

The remote computer system can repeat this process for other selectrooms on select floors in select buildings on the site in order topopulate a set of room layers (or “space models”) representing uniqueconstellations of features that are and/or may be present in these roomsat any given time. The remote computer system can then compile theseroom layers, floor layers, building layers, and site layer into the sitemap for the site and store this site map in a site map database.

4.2 Variations

In one variation, the remote computer system can: access architecturaland/or engineering plans of the site, buildings on the site, and floors(or “levels”) of these buildings from the administrator via anadministrator portal, such as described above; and then prompt theadministrator to select buildings, floors, or specific rooms of interest(e.g., buildings, floors, or rooms in which equipment linked tovirtual-assist protocols is present). The remote computer system canthen implement methods and techniques similar to those described above:to generate a geospatial map of the site and various buildings withinthe site; to link available wireless site surveys to discrete buildingsor specific floors containing at least one room of interest selected bythe administrator; and to link each room of interest selected by theadministrator to a space model, as described above.

The remote computer system can compile the geospatial map of the site,wireless site surveys for specific buildings or floors, and space modelsfor specific rooms of interest into a site map for the site.

However, the remote computer system can generate and store a site mapcontaining any other representation of the site, buildings on the site,floors within these buildings, and/or rooms of interest within thesefloors in any other form or format.

5. LOCALIZATION

The remote computer system and the mobile device can then cooperate todetermine a coarse location of the mobile device in Block S112 based ongeospatial data collected by the mobile device in Block S110, todetermine a location of the mobile device with moderate granularity inBlock S122 based on wireless connectivity data collected by the mobiledevice in Block S120, and to then determine a fine location (or “pose”)of the mobile device in Block S134 based on optical data recorded by themobile device and a space model loaded onto the mobile device in BlockS130, as shown in FIGS. 3 and 5 .

5.1 Global Localization: Geospatial

During operation, upon receipt of a geospatial location from a mobiledevice carried or worn by an operator, the remote computer system canquery the site map database for a particular site map containing a sitelayer defining a site boundary that contains the geospatial location ofthe mobile device and retrieve this particular site map accordingly. Theremote computer system can then scan the site layer in the particularsite map for a particular building nearest (or containing) thegeospatial location of the mobile device. If the remote computer systemthus identifies a particular building containing or falling within athreshold geospatial distance of the geospatial location of the mobiledevice, the remote computer system can predict that the mobile device isoccupying the particular building and then extract the building layerand related floor layers for this particular building from theparticular site map accordingly.

(The remote computer system can additionally or alternatively predictthe mobile device is occupying the particular building if a uniqueidentifier of a wireless AP received by the mobile device matches aunique identifier of a known wireless AP in the particular building,such as stored in the building layer or in a floor layer for theparticular building.)

5.2 Local Localization: Wireless Access Points

Once the remote computer system has thus isolated the mobile device to aparticular building on the site, the remote computer system can: querythe mobile device for a set of unique identifiers of wireless APscurrently within wireless range of the mobile device and compare the setof unique identifiers received from the mobile device to uniqueidentifiers of wireless APs located on floors of the particularbuilding—as represented in a set of floor layers for the particularbuilding—in order to isolate the mobile device to a particular floor ofthe particular building. The remote computer system can then return aspace model for each room of interest on the particular floor to themobile device for further localization of the mobile device.

Alternatively, if the particular floor includes multiple wireless APsand the set of unique identifiers received from the mobile devicecorresponds to only a subset of these wireless APs, the remote computersystem can further isolate the mobile device to a particular subregionof the particular floor in which the wireless APs detected by the mobiledevice are present. The remote computer system can then aggregate spacemodels for a particular subset of rooms in this particular subregion ofthe particular floor and return space models for this particular subsetof rooms to the mobile device for further localization of the mobiledevice.

The remote computer system can additionally or alternatively query themobile device for strengths of wireless signals received from wirelessAPs that it recently detected. Based on these signals strengths, knownlocations of wireless APs on the particular floor, and known wirelesscharacteristics of these wireless APs, the remote computer system cantriangulate the particular location of the mobile device within theparticular floor. For example, the remote computer system can implementRSSI techniques to triangulate the location of the mobile devicerelative to multiple wireless APs on the particular floor with toleranceof +/−five meters. Similarly, in the implementation described above inwhich the remote computer system accesses wireless site surveyscontaining 2D or 3D maps (or gradients) of wireless signal strengthsfrom multiple wireless APs on floors of buildings on the site and storesthese maps of wireless signal strengths in floor layers for floors ofthese buildings, the remote computer system can scan the map of wirelesssignal strengths stored in the floor layer of the particular floor inorder to isolate a particular location likely to be occupied by themobile device. However, the remote computer system can implement anyother method or technique to calculate or estimate a more specificlocation (or area, subregion) of the mobile device within the particularfloor.

Thus, if the remote computer system determines that the mobile device isoccupying a hallway, the remote computer system can: scan the floorlayer to identify a small number of (e.g., up to four) rooms of interestnear the mobile device; aggregate space models for each of these rooms;and return these space models to the mobile device for furtherlocalization of the mobile device. Similarly, if the remote computersystem thus determines that the mobile device is occupying a particularroom but with a loose spatial tolerance, the remote computer system can:calculate a localization window in which the mobile device is likely tobe located; scan the floor layer to identify other rooms of interestthat intersect the localization window; aggregate space models for theparticular room and these other rooms of interest; and then return thesespace models to the mobile device for further localization of the mobiledevice.

Alternatively, if the remote computer system thus determines that themobile device is occupying a particular room with a tight spatialtolerance or otherwise calculates a localization window that intersectsonly one room of interest, the remote computer system can transmit thespace model for this specific room of interest to the mobile device forfurther localization of the mobile device. Yet alternatively, the remotecomputer system can upload a space model for a particular room to themobile device only after determining that the location of the mobiledevice—determined based on wireless connectivity data collected by themobile device—intersects a room of interest or intersects a roomoccupied by an equipment unit scheduled for a test or operationalprotocol.

In one variation, the user manually indicates the particular buildingand/or particular space or room she is occupying, such as by manuallyindicating this space in a virtual map of the site rendered on thedisplay of the mobile device, by selecting this space from a dropdownmenu rendered on the mobile device, or orally stating her location tothe mobile device. The mobile device can then automatically retrieve aspace model for the indicated location of the mobile device, such asdescribed above.

5.4 Local Localization: Wireless Access Points+Optical Features

In one variation, the remote computer system isolates a particular roomor space occupied by the mobile device based on a combination ofwireless connectivity data and image data collected by the mobiledevice.

In one implementation, the mobile device generates a list of identifiersand signal strengths of wireless networks accessible to the mobiledevice and returns these to the remote computer system. The remotecontroller then implements the foregoing methods and techniques toestimate the mobile device's location, such as by: isolating aparticular building occupied by the mobile device based on knownlocations of wireless APs on the list received from the mobile device;and triangulating the mobile device's location based on signal strengthsof the wireless networks and a map or table of geospatial locations ofthese wireless APs. However, this triangulated location of the mobiledevice may be relatively inaccurate due to multi-path delays,reflections, and obstructions within the space occupied by the mobiledevice. Therefore, the mobile device can also transmit a raw image or adescriptor of a set of features extracted from an image recorded by themobile device to the remote computer system, and the remote computersystem can verify the location of the mobile device by comparingfeatures extracted from this image to individual space models for thediscrete spaces around the triangulated position of the mobile device orto a global space model for the floor or building occupied by the mobiledevice.

For example, the remote computer system can: isolate the mobile deviceto one floor or cluster of spaces in a building based on identifiers andsignal strengths of wireless APs accessible to the mobile device; andthen retrieve a set of discrete space models (e.g., 3D localizationmaps) representing discrete spaces on the floor of the building orrepresenting each space in this cluster of spaces. The remote computersystem can also query the mobile device for an image (e.g., 2D colorimage, a 3D depth map or point cloud); or the mobile device can returnthis image to the remote computer system automatically with the list ofwireless APs. For example, the mobile device can opportunisticallycapture an image when the camera on the mobile device is not obscured,when the mobile device is arranged in an approximately uprightorientation (e.g., such that the camera is approximately normal to anearest wall), and when the mobile device detects limited motion (e.g.,such that the resulting image exhibits minimal blur); the mobile devicecan then return this image to the remote computer system for refinedlocalization. In another example, the mobile device can record asequence of images (e.g., a “video clip”), select a particular image inthis sequence that exhibits least blur and/or depicts a greatest numberof distinct features, and return this image to the remote computersystem.

Upon receipt, the remote computer system can detect and extract aconstellation of features from this image, such as text, iconography,planes, or visual patterns arranged on large surfaces (e.g., floors andwalls) depicted in the image. (Alternatively, the mobile device canrecord and return a video clip to the remote computer system, and theremote computer system can extract features from multiple frames in thisvideo clip. Yet alternatively, the mobile device can extract thesefeatures from the image and return a container—such as a matrix orvector describing these extracted features—to the remote computersystem.)

The remote computer system can then: scan the set of space models forlike or similar features, such as similar text, iconography, or relativepositions of planes; and isolate a particular space model—from this setof space models—that contains a greatest number of like or similarfeatures. Upon isolating this particular space model, the remotecomputer system can implement 2D or 3D localization techniques tocalculate a position of the mobile device within a particular spacecorresponding to this particular space model. For example, the remotecomputer system can: calculate a transform that aligns the constellationof features extracted from the image to a constellation of like featuresrepresented in the particular space model; interpret an initial location(i.e., a position and orientation, or “pose”) of the mobile device inthe particular space based on this transform; and return the particularspace model and the initial location of the mobile device to the mobiledevice to enable the mobile device to rapidly localize itself within theparticular space without necessitating transmission of data to theremote computer system.

Alternatively, the remote computer system can: retrieve a global modelof the building floor associated with wireless APs detected by themobile device; scan the global model for the set of features extractedfrom the image; localize the mobile device within the global model ofthe building floor; and then return a segment of the global model aroundthe mobile device's location (e.g., a predefined space model for aparticular space containing the mobile device's location, a segment ofthe global model extracted ad hoc by the remote computer system for themobile device) and the initial location of the mobile device to themobile device.

The mobile device can then implement methods and techniques describedbelow to localize the mobile device with a high degree of granularity orresolution by: recording a second image; extracting a secondconstellation of features from this second image; implementingdead-reckoning techniques to estimate an offset from the initiallocation received from the remote computer system based on motion of themobile device between recordation of the first and second images;predicting a second location of the mobile device at the current timebased on the sum of the initial location and the offset; scanning thelocal copy of the space model—around the predicted second location ofthe mobile device—for the second constellation of features; calculatinga second transform that aligns this second constellation of featureswith a corresponding constellation of like features in the space model;and then calculating and storing this image-based second location andorientation of the mobile device relative to the space based on thissecond transform. The mobile device can repeat this process forsubsequent images recorded by the mobile device, such as until the userdisables localization at the mobile device, completes a protocol, orexits the particular space, at which time the mobile device may nolonger match features detected in images to features represented in thespace model and can thus revert to interfacing with the remote computersystem to localize the mobile device based on connectivity to wirelessAPs.

Therefore, in this variation, the remote computer system can: detect afirst set of optical features in a first image recorded by the mobiledevice at a first time; estimate occupation of the mobile device in acluster of spaces within a particular building at the first time basedon a list of identifiers of wireless APs detected by the mobile deviceand a map of wireless network connectivity within the particular site;compare the first set of optical features to constellations of referencefeatures represented in a set of space models corresponding to thecluster of spaces; and confirm presence of the mobile device in aparticular space—in the cluster of spaces in the particular building—atthe first time based on alignment between the first set of opticalfeatures and the constellation of reference features represented in thespace model of the particular space. The remote computer system can thenreturn—to the mobile device—a relatively small space model depictingfeatures in the particular space most likely to fall in the field ofview of the optical sensor in the mobile device, thereby: limitingbandwidth requirements to load localization data onto the mobile device;limiting a file size cached by the mobile device to support localizationservices; and enabling the mobile device to perform rapid localizationprocesses locally regardless of wireless connectivity while the mobiledevice remains in or near this particular space.

(Additionally or alternatively, the remote computer system can thenreturn—to the mobile device—a relatively small space model depictingwireless signal strengths of local wireless transmitters located on ornear equipment units in the particular space, thereby: limitingbandwidth requirements to load localization data onto the mobile device;limiting a file size cached by the mobile device to support localizationservices; and enabling the mobile device to perform rapid localizationprocesses locally regardless of connectivity to a wireless access pointwhile the mobile device remains in or near this particular space.)

5.5 Global and Local Localization Variation

In one variation, the mobile device: stores the site, building, and/orfloor layers of the site map; locally executes the forgoing methods toestimate the location of the mobile device with moderate granularitybased on wireless connectivity data collected by the mobile device; andreturns a request for a space model of a particular room to a remotespace model database, such as if a space model is available for theparticular room or if the particular room is occupied by an equipmentunit scheduled for a test or operational protocol, as described above.

The mobile device and/or the remote computer system can regularly repeatthe foregoing methods and techniques to track the location of the mobiledevice with moderate granularity and to selectively upload a space modelto the mobile device accordingly.

5.6 Hyper-Local Localization In Space Coordinate System: OpticalFiducials

Following receipt of a space model (e.g., in the form of a 3Dlocalization map) of a particular room occupied by the mobile device,the mobile device can compare optical data read by optical sensors inthe mobile device (or in the separate augmented reality device) and thespace model to calculate a pose of the mobile device (or the augmentedreality device) in real space relative to immutable features in thisroom. In one implementation, the mobile device regularly: samplesoptical sensors integrated into the mobile device (or integrated into aconnected augmented reality device); compiles concurrent images read bythese optical sensors into a 3D point cloud; and implementspoint-to-plane fitting or other localization techniques to calculate atransform that maps this 3D point cloud onto the space model (e.g., thelocalization map of the room). This transform can thus represent a poseof the mobile device within the room, and the mobile device can leveragethis transform to locate spatial, augmented reality content on theheads-up display of the augmented reality device.

Alternatively, the mobile device can: detect a 2D constellation offeatures representing a set of active or passive optical fiducials inthe space from a 2D image recorded by an optical sensor in the mobiledevice; calculate a scale, warp, and orientation that projects this 2Dconstellation of features onto representations of these opticalfiducials in the space model; and then derive the location andorientation of the mobile device in the space based on this scale, warp,and orientation.

However, the mobile device can implement any other image-basedlocalization technique to transform optical data recorded by the mobiledevice (or by a connected device) into a location of orientation (or“pose”) of the mobile device in the room. The mobile device can repeatthese methods and techniques for subsequent images recorded by themobile device in order to track the pose of the mobile device over time.

5.7 Example

Therefore, the remote computer system and the mobile device cancooperate to localize the mobile device within a particular room orspace in a particular building at a particular site. In one example, asa user enters a job site (e.g., by pulling in to a parking lot at herplace of employment), the mobile device can calculate a first geospatialcoordinate—in a geospatial coordinate system—of a first location itoccupies at a first time and then transmit this first geospatialcoordinate and a first query for location services to the remotecomputer system. The remote computer system can then locate the mobiledevice at this site at the first time in response to the firstgeospatial coordinate received from the mobile device intersecting ageoreferenced boundary associated with this site. As the user walks fromthe parking lot into a building on the site, such as to begin a test,experiment, or other procedure at an equipment unit in this building,the mobile device can: aggregate a first list of identifiers of a firstset of wireless APs within wireless range of the mobile device at asecond, subsequent time; and then transmit this first list ofidentifiers and a second query for location services to the remotecomputer system. Accordingly, the remote computer system can estimateoccupation of the mobile device in a particular room or space in aparticular building on the site based on this first list of wireless APidentifiers and a map or table of wireless network connectivitythroughout the site. Once the remote computer system thus identifies theparticular space occupied by the mobile device at the second time, themobile device can download a space model of the particular space (likelyto be) occupied by the mobile device. For example, the space model caninclude a 3D localization map depicting positions of immutable objectsand surfaces in the space, such as walls, floor, ceilings, doors, andpartitions. Alternatively, the space model can include a 3D localizationmap depicting positions of active and/or passive optical fiducials, suchas: quick-response and barcodes applied to walls and floors; illuminatedexit signage near doors; retro-reflector strips applied to referencesurfaces; light-absorbing strips applied to reference surfaces; ceilinglighting; and/or dedicated, active optical fiducials containing activelight elements.

With the space model for this particular space thus loaded onto themobile device, the mobile device can: access an image recorded by theoptical sensor in the mobile device at a third time; extract a set ofoptical features (e.g., planes depicting walls and floors; geometries ofreflective or absorptive surfaces; positions of active light elements)from this color image; and calculate a position of the mobile devicewithin the particular space at approximately the third time based onpositions of the first set of optical features relative to aconstellation of reference features represented in the 3D localizationmap. In particular, the mobile device can derive its location andorientation in the particular space by aligning a 3D constellation offeatures represented in the space model to: features extracted from asingle 2D color image recorded by a color camera in the mobile device;3D features extracted from a depth map or 3D point cloud recorded by adepth sensor in the mobile device; or 3D features extracted from asynthetic 3D point cloud generated or derived from a sequence of 2D(color) images recorded by a 2D camera.

In this example, the mobile device can also triangulate its locationwithin the particular space based on signal strengths of wirelesssignals received from local, short-range wireless transmitters locatedon or near equipment units in the space based on a map of wirelesssignal strengths of these wireless transmitters stored in the spacemodel. The mobile device can then verify the image-based location thuscalculated based on this wireless-based location. Furthermore, duringsampling intervals in which the mobile device fails to detect referencefeatures in the field of view of the camera (e.g., when the mobiledevice is placed face-up on a table such that the camera is obscured),the mobile device can transition to tracking its location within theparticular based on wireless signal strengths of these local,short-range wireless transmitters.

(Alternatively, the mobile device can execute high-resolutionlocalization techniques based on wireless connectivity (e.g., signalstrength) to these local, short-range wireless transmitters exclusive ofoptical data. More specifically, in this variation, the mobile devicecan thus execute methods and techniques described below based on thesewireless connectivity to local, short-range wireless transmitters ratherthan optical data captured by an optical sensor in the mobile device.)

However, the mobile device can implement any other method or techniqueto calculate its location and orientation in the particular space basedon the space model and optical features detected in a field of view ofan optical sensor integrated into or connected to the mobile device.

6. PROTOCOL SELECTION

If the mobile device thus determines its location in the particular roombased on the space model and optical data thus recorded by opticalsensors integrated into the mobile device (or into the separateaugmented reality device), the mobile device (or the remote computersystem) can automatically identify a particular protocol that iscurrently pending for an equipment unit nearby and automatically loadthis protocol in Block S140, as shown in FIGS. 3 and 5 .

For example, the remote computer system can store a set ofvirtual-assist protocols in a database. Each protocol: can be associatedwith a particular equipment unit or a particular type or class ofequipment unit on the site; can include a set of instructions for inputinto this equipment unit, such as in the form of textual and graphicalaugmented reality overlays spatially referenced to input regions on theequipment unit; and can include a set of instructions for data capturefrom this equipment unit, such as in the form of textual and graphicalaugmented reality overlays spatially referenced to data output regionsof the equipment unit. When a protocol is loaded into a mobile devicewhile the mobile device is present near a corresponding equipment unit,the mobile device can render—in the augmented reality environment—asequence of instructions for input into and data capture from theequipment unit, wherein these instructions are spatially aligned toreference features on the equipment unit in the field of view of anoperator and corresponding to a sequence of steps in the protocol, inorder to provide real-time, strategic guidance to the operator whiletesting or operating the equipment unit, thereby reducing opportunityfor operator error, increasing operator efficiency, and reducing needfor a second operator to oversee the operator while executing thisprotocol.

In one implementation, once the mobile device determines its pose withinthe particular room in Block S134, the mobile device (and/or the remotecomputer system): queries the floor layer, the building layer, or anengineering plan of equipment distribution throughout the building for atype and/or unique identity of a particular equipment unit nearest themobile device; scans a protocol schedule for this particular equipmentunit for a next scheduled protocol for the particular equipment unit;and serves a prompt to an operator carrying the mobile device and/orwearing the augmented reality device to confirm this protocol. Themobile device can then load this protocol once confirmed by theoperator.

In a similar implementation, the mobile device (and/or the remotecomputer system): queries the floor layer, the building layer, or theengineering plan of equipment distribution throughout the building fortypes and/or unique identities of a cluster of equipment units near themobile device; scans protocol schedules for these particular equipmentunits for a next scheduled protocol for a particular equipment unit inthis cluster; renders a list of these protocols—such as ranked by timeproximity to scheduled completion—on heads-up display; and serves aprompt to the operator to select a protocol in this list. The mobiledevice can then load a particular protocol selected by the operator andrender augmented reality cues on the heads-up display to engage thecorresponding equipment unit.

In a similar implementation, the mobile device (or the remote computersystem) can filter protocols—queued for equipment units near the mobiledevice—based on whether the user is qualified to operate these equipmentunits or otherwise holds qualifications to perform these protocols. Forexample, in response to identifying the particular space occupied by themobile device, the mobile device (or the remote computer system) can:access a user profile associated with the user currently assigned orlogged into the mobile device; and read a list of specific protocols forwhich the user is prequalified to perform and/or a list of equipmentunits the user is qualified to operate from the user's profile.Accordingly, the mobile device (or the remote computer system) can:aggregate a list of protocols queued for equipment units located in theparticular building occupied by the user; filter this list of protocolsto include only protocols the user is qualified to perform and onequipment units that the user is qualified to operate; and then presentthis list of protocols to the user, such as by rendering this filteredlist of protocols on a display integrated into or connected to themobile device. In this example, the mobile device can also sort thisfiltered list of protocols based on proximity of the mobile device toequipment units associated with protocols on this filtered list. As theuser moves throughout the space—such as past different equipment unitsof the same or different equipment types—the mobile device can implementthe foregoing methods and techniques to: recalculate the location or themobile device; query a map of the building, floor, or space currentlyoccupied by the mobile device for equipment units near the mobiledevice's location; query a scheduler or database for protocols queuedfor these equipment units; filter this revised list of protocols basedon the user's qualifications; re-sort this filtered list of protocolsbased on proximity of corresponding the equipment units to the mobiledevice's current location; and prompt the user to select from thisre-sorted list of protocols rendered on the mobile device or connecteddisplay.

In response to selection of a particular protocol from the list ofprotocols thus presented to the user, the mobile device can download theparticular protocol from a remote database or access this particularprotocol from a local cache or local memory.

However, the mobile device can implement any other method or techniqueto automatically load a protocol for a particular equipment unit or toprompt the operator to elect a particular protocol from a small set ofprotocols pending for equipment units near the mobile device's currentlocation. The mobile device can then initiate a new instance of theprotocol at the mobile device.

7. HYPER-LOCAL LOCALIZATION IN MACHINE COORDINATE SYSTEM

In one variation shown in FIG. 5 , once the user (or the mobile device)selects a protocol and initiates a new instance of the protocol for aparticular equipment unit in the space, the mobile device can retrievean equipment model for a type of the particular equipment unit, such asby downloading a 3D model of the particular equipment unit from theremote computer system or other remote database. The mobile device canthen implement the foregoing methods and techniques to localize themobile device to the particular equipment unit based on the equipmentmodel—rather than or in addition to localizing the mobile device to thespace based on the space model. In particular, the equipment unit mayhave moved within the space since the space model was created or lastupdated. Therefore, the mobile device can implement the equipment modelto localize the mobile device to the particular equipment unit (e.g.,based on optical features detected in the field of view of the opticalsensor in the mobile device) rather than to the space model, therebyenabling the mobile device to reject localization error that mayotherwise occur due to past movement of the equipment unit in the spaceand to render augmented reality guidance—for steps in theprotocol—aligned to corresponding features (e.g., input controls,displays) on the equipment unit with a high degree of precision.

For example, once the mobile device accesses the equipment model andinitiates the current instance of the protocol, the mobile device can:detect a set of optical features in the field of view of the opticalsensor (e.g., by extracting 2D features from one image, by extracting 3Dfeatures from one depth map, or by deriving 3D features from a sequenceof 2D images recorded by the optical sensor); and calculate a positionof the mobile device relative to the equipment unit based on positionsof this set of optical features relative to a constellation of referencefeatures represented in the equipment model of the equipment unit. Themobile device can then serve augmented reality guidance—spatiallyreferenced to the equipment unit—for steps of the protocol. Additionallyor alternatively, the mobile device can selectively open steps of theprotocol (e.g., instructions for input control values; capture pathwaysfor recording numerical, textual, audible, or image-based data from theequipment unit) only after localizing the mobile device to locationswithin threshold distances of control inputs, displays, or otherreferences locations on the equipment unit associated with these steps.

In this variation, once the mobile device access the equipment model,the mobile device can calculate its location and orientation relative tothe equipment unit by default; however, if the mobile device detects aninsufficient number or distribution of features—in a video feed recordedby the optical sensor—depicting the equipment unit, the mobile devicecan transition to localizing the mobile device to the space model, suchas when the user turns her back to the equipment unit or faces aworktable near the equipment unit during the protocol. Alternatively,the mobile device can calculate its location and orientation relative tothe space based on the space model by default but can transition tolocalizing itself to the equipment unit during select steps of theprotocol associated with augmented reality content spatially-referencedto the equipment unit.

8. HYPER-LOCAL LOCALIZATION: STATIONARY MOBILE DEVICE

In one variation, the mobile device executes the foregoing processes: toextract features from images (or frames, depth maps) recorded by theoptical sensor as the mobile device moves throughout the space (e.g., asthe user walks through the space, around the equipment unit, and betweenthe equipment unit and support stations in the space); and to localizethe mobile device in the space (or relative to the equipment unit) basedon these features. However, motion of the mobile device during thisprocess may produce blur in these images, which may increaselocalization error when the mobile device calculates its position in thespace (or relative to the equipment unit) based on comparisons ofconstellations of blurred features to reference features represented inthe space model (or in the equipment model).

Therefore, for steps of the protocol containing augmented realitycontent flagged (or “tagged”) for high spatial accuracy, the mobiledevice can prompt the user to cease motion and/or hold the mobile devicein one approximately static location, thereby enabling the mobile deviceto capture images with low blur, extract crisp features from theseimages, and localize the mobile device to the space (or to the equipmentunit) with high spatial accuracy based on these crisp features. Forexample, the mobile device can execute this process to prompt the userto cease motion in preparation for: highlighting a particular switch ina cluster of switches on a control panel; highlighting a particularnumerical value to read from a particular location within a string ofvalues on a display; or highlighting a narrow target range of acceptableneedle positions on an analog dial.

In one implementation, after loading a step in the protocol associatedwith a particular element on the equipment unit (e.g., a switch, a valueon a display, a dial), the mobile device can: serve a prompt to the userto cease motion for a period of time; access an image (e.g., a colorframe, a depth map, a 3D point cloud) recorded by the optical sensorduring this period of time; extract a set of optical features from thisimage; calculate a second transform that projects this set of opticalfeatures onto corresponding reference features in the equipment model(or in the space model) with reduced error (i.e., compared tolocalization based on images recorded during motion of the mobiledevice); and calculate a position of the mobile device—at increasedresolution (i.e., compared to localization based on images recordedduring motion of the mobile device) relative to the equipment unit (orwithin the space) during this period of time. The mobile device can thenrender guidance for this step of the protocol—visually aligned with theparticular element on the equipment unit—such as on a display integratedinto or connected to the mobile device.

For example, the mobile device can: implement computer vision techniquesto identify the particular element—defining a particular control inputin a cluster of control inputs at a control panel on the equipmentunit—in this image; extract a location of the particular element fromthe image; and calculate a position of the particular element in thefield of view of a viewfinder rendered on a 2D display integrated intothe mobile device or rendered on a heads-up display coupled to themobile device, such as based on a known offset between the opticalsensor that captured the image and the display. The mobile device canthen: generate an augmented reality frame containing a visual object(e.g., a color circle) distinguishing the particular control input inthe cluster of control inputs and visually aligned with the particularcontrol input when viewed on the 2D or heads-up display; and render theaugmented reality frame on the 2D or heads-up display. Subsequently, themobile device can: prompt the user to perform the indicated stepsaccording to the guidance, including moving the mobile device from itscurrent location; implement dead-reckoning and the preceding methods andtechniques to track the location of the mobile device relative to theparticular element on the equipment unit as the mobile device moves awayfrom this derived high-resolution location.

Alternatively, the mobile device can implement the foregoing methods andtechniques to localize the mobile device based on optical data recordedwhile the mobile device is in motion near low-risk zones in the space.However, as the mobile device approaches a high-risk zone in the space(e.g., a fall- or slip-prone area, a high-temperature area, an openchemical storage or processing location, a high-radiation area) orprepares to access a high-risk step in the protocol, the mobile devicecan prompt the user to cease motion of the mobile device, captureoptical data while the mobile device is (approximately) static, andlocalize the mobile device in the space (or relative to the equipmentunit) with increased resolution and reduced error based on crispfeatures extracted from images recorded by the mobile device during thisperiod. The mobile device can subsequently leverage thishigh-resolution, low-error location of the mobile device while static tomaintain relatively high-resolution, relatively low-error localizationof the mobile device as the mobile device moves through this high-riskzone or through this high-risk step.

8. MOBILE EQUIPMENT AND SPACE CONFIGURATION

In one variation described above, the mobile device loads a space modelthat includes 2D models, 3D models, template images, or otherrepresentations of mobile equipment and other objects commonly presentin the particular room, floor, building, or site and/or specified inpending protocols for equipment units in the particular room occupied bythe mobile device. In this variation, the mobile device can executecomputer vision methods and techniques—such as template matching, objectdetection, and/or object recognition—or artificial intelligencetechniques to identify mobile equipment and other mutable objects nearthe mobile device. The mobile device can then construct a map, table, orother representation of the space it occupies, including types andlocations of these mobile equipment and other mutable objects.

The mobile device can then write this map, table, or otherrepresentation of the configuration of the space to a protocol recordfor the current instance of the protocol thus initiated at the mobiledevice. Throughout the protocol, the mobile device can continue todetect these mobile equipment and other objects in the space—such asrelative to the space, room, or target equipment unit associated withthe protocol active at the mobile device—and write additionaltimestamped maps, tables, or other representations of the distributionof mobile equipment and other objects in the space to this protocolrecord, such as continuously throughout this instance of the protocol(e.g., once per second) or when the mobile device detects that thesemobile equipment and objects have moved in the space.

Furthermore, once the operator confirms a particular protocol, themobile device can confirm that mobile equipment specified in theparticular protocol is present at or near the corresponding equipmentunit. For example, the mobile device can: implement methods andtechniques described above to construct a map, table, or otherrepresentation of the room, including types and locations of mobileequipment and other mutable objects in the field of view of the mobiledevice; and then compare this map, table, or other representation of theroom to a list of objects specified by the particular protocol toisolate any missing equipment, materials, or other objects. The mobiledevice can then render a prompt—via the heads-up display—to retrieve anymissing equipment, materials, or other objects before initiating theelected protocol.

9. PROTOCOL GUIDANCE

Once the protocol for the equipment unit is selected, verified, andloaded onto the mobile device, the mobile device can render a sequenceof instructions in the protocol within the augmented realityenvironment, thereby enabling the operator to view real-timeinstructions spatially aligned to features on the equipment unit thatrelate to these instructions.

9.1 Device Position and Protocol Guidance

Generally, in Block S142, the mobile device can implement methods andtechniques described above to track its positions in real space based onoptical data collected by the optical sensor(s) and based on thelocalization map for the room. For example, for each subsequent set ofconcurrent frames recorded by a set of optical sensors in the mobiledevice, the mobile device can: compile this set of concurrent framesinto a sparse 3D point cloud; and calculate a sensor transform thataligns features in the sparse 3D point cloud to surfaces represented inthe localization map of the room, such as to surfaces that represent theequipment unit specifically in the localization map. This sensortransform can thus represent the pose of the optical sensors in realspace relative to the room or to the equipment unit more specifically.The mobile device can then apply a sensor-to-display transform—thatrepresents a known offset between the optical sensors and the heads-updisplay—to the sensor transform to calculate a display transform thatrepresents the pose of the optical sensors in real space relative to theroom or to the equipment unit more specifically. The mobile device caninject augmented reality content (e.g., spatially-referenced textualand/or graphical instructions) into a next (augmented reality) framebased on this display transform in order to align this content to theequipment unit in the operator's field of view.

Alternatively, the protocol can specify registration of a particularinstruction to a particular feature on the equipment unit. Duringexecution of the protocol, the mobile device can thus: select a nextinstruction within the protocol; scan an image recorded by the opticalsensor(s) for a feature linked to this next instruction; extract a poseof the optical sensor relative to this feature (or vice versa) from theimage; and locate the instruction in a next augmented reality framebased on the pose of the optical sensor relative to this feature and aknown offset from the optical sensor to the heads-up display.

However, the mobile device can implement any other method or techniqueto locate augmented reality content—for the current instruction in theprotocol—within an augmented reality frame. The mobile device can thenrender this augmented reality frame on the heads-up display in (near)real-time and then repeat this process to inject augmented realitycontent for the current instruction in subsequent augmented realityframes based on each subsequent image recorded by the optical sensor(s)as the operator moves through real space.

The mobile device can then index to a next instruction in the protocoland repeat the foregoing methods and techniques responsive to manualconfirmation by the operator that the current step of the protocol hasbeen completed. Alternatively, the mobile device can: implement computervision and/or artificial intelligence techniques to automatically detectthat the current step of the protocol has been completed and index to anext step accordingly; or automatically index to the next step of theprotocol after automatically recording data from the equipment unitaccording to the current step, as described below. The mobile device canthen repeat the foregoing methods and techniques to renderspatially-located instructions for the next steps of the protocol—in theaugmented reality environment—until the protocol is completed.

9.2 Data Capture

In one variation, the mobile device automatically records data from theequipment unit during execution of the protocol by the operator. In oneimplementation, a particular step of the protocol specifies automatedrecordation of a value from a particular display on the equipment unit(e.g., an analog dial or gauge, a digital display). In thisimplementation, the space model (e.g., the localization map) can specifya position of this display on the equipment unit, and the mobile devicecan: calculate its pose in real space relative to the equipment unitbased on features detected in an image recorded by the optical sensorand the space model; and isolate a region of the image corresponding tothe display based on the pose of the mobile device and the location ofthe display specified in the space model. Alternatively, the mobiledevice can detect the display directly in this image, such as bydetecting a reference optical fiducial located on the equipment unitimmediately adjacent the particular display or by implementingtemplate-matching techniques to isolate a region of the image thatrepresents the display. The mobile device can repeat this process foreach subsequent image recorded by the optical sensor during this step ofthe protocol in order to track the position of display relative to themobile device.

When the mobile device determines that the position of the opticalsensor falls within a threshold distance (e.g., one meter) and within athreshold angular offset (e.g., 15° in each dimension) of the displaybased on features detected in a current image recorded by the opticalsensor, the mobile device can extract a value from a region of thisimage corresponding to the display. For example, the mobile device can:isolate a region of this image corresponding to the display; implementoptical character recognition and/or other computer vision techniques toextract a value from this region of the image; and write this value toan electronic lab notebook or other data repository associated with theequipment unit and/or with current execution of the protocol.

In this variation, the mobile device can also calculate a quality ofthis image (e.g., a resolution of the region of the image correspondingto the display, a sharpness of the image) and/or a confidence score thatthe value extracted from this region of the image is accurate. If thequality of the image is less than a threshold quality or if theconfidence score is less than a threshold score, the mobile device canrender—on the heads-up display—a prompt for the operator to move closerto the display to enable automated documentation of this step in theprotocol. Alternatively, if the quality of the image is less than athreshold quality or if the confidence score is less than a thresholdscore, the remote computer system can repeat the foregoing process toopportunistically record a second image as the operator moves around theequipment unit, to extract data from a region of the second imagerepresenting the display, and to quantify a quality of the second imageand/or data extracted from this second image as the operator performsthe current step of the protocol. If the quality of the second imageand/or the confidence score for data extracted from the second imageexceeds that of the previous image, the remote computer system canoverride data extracted from the previous image with data extracted fromthe second image. The remote computer system can repeat this processuntil the current step of the protocol is completed, until the qualityof the last image recorded exceeds the threshold quality, and/or untilthe confidence score for data extracted from the last image exceeds thethreshold score.

In this variation, this step of protocol can additionally oralternatively specify manual recordation of a value from the display,and the mobile device can: implement similar methods and techniques tocalculate the location of the display relative to the heads-up display;generate an augmented reality frame containing a highlighted (e.g., a“green”) boundary that encompasses the display on the equipment unitwhen the augmented reality frame is rendered on the heads-up display;populate the augmented reality frame with an instruction to record avalue from this display; and then render this augmented reality frame onthe heads-up display in (near) real-time in order to prompt the operatorto manually record data from the display. The mobile device can repeatthis process until the operator manually confirms that this step hasbeen completed or until the mobile device automatically determines thatthis step is complete.

9.3 Protocol Observation

A step in the protocol can also specify recordation of video (or a stillimage) during performance of this step. Accordingly, the mobile devicecan: automatically record a video (or a still image) during execution ofthis step by the operator and append a report for this instance of theprotocol with this video (or still image). A second remote operator oradministrator can access this video in real-time in order to manuallyverify that this step was performed properly. Alternatively, the secondoperator portal or administrator can review this video hours, days, orweeks following conclusion of the protocol in order to verify this stepof the protocol.

However, the mobile device can implement any other method or techniqueto automatically record values from a display on the equipment unit, torecord a particular step of the protocol, or to record the protocol inits entirety during execution by the operator.

9.4 Gated Protocol Steps

In one variation shown in FIG. 5 , the mobile device limits (or “gates”)access to a step in the protocol based on location of the mobile device.For example, for an action step specifying an input (e.g., selection ofa control input, adjustment of a valve, supply of material) at aparticular location at the equipment unit or in the space, the mobiledevice can prevent confirmation of this action step and/or preventaccess to a next step in the protocol until the mobile device determinesthat its location falls within a threshold distance (e.g., one meter; asum of 50 centimeters and a localization tolerance of the mobile deviceand space model) of the particular location associated with this actionstep. Similarly, for a data capture step specifying recordation of data(e.g., in the form of a manually-input value, automatic data capture bythe mobile device, audio capture, image capture, or video capture) at aparticular location at the equipment unit or in the space (e.g., at alocation of a dial or screen displaying data designated for capture inthis step), the mobile device can prevent manual entry of data, opticalor audio captured triggered by the user, or automatic data captured bythe mobile device until the mobile device determines that its locationfalls within a threshold distance of the particular location associatedwith this data capture step.

Therefore, the mobile device can regularly execute the foregoing methodsand techniques to recalculate the location of the mobile devicethroughout the protocol and selectively enable the user to open a nextstep in the protocol, confirm completion of a step, supply dataspecified in a step, and/or transition to a next step based on whetherthe mobile device has moved within a threshold distance of locations inthe space (or relative to the equipment unit) associated with thesesteps in the protocol.

10. PROTOCOL COMPLETION+SPACE DEPARTURE

In one variation, the mobile device discards the space model (and or theequipment model) when the mobile device determines that it has exitedthe space or moved more than a threshold distance from the equipmentunit following completion or closure of the current instance of theprotocol.

In one implementation, the mobile device: downloads the space model forthe particular space once the remote computer system determines that themobile device is occupying the particular space, as described above; andtracks positions of the mobile device within the particular space basedon positions of optical features detected in the field of view of theoptical sensor relative to reference features represented in the spacemodel over a period of time (e.g., during the current instance of theprotocol). However, upon conclusion of this instance of the protocol (orafter the user pauses or cancels this instance of the protocol), themobile device may: detect absence of reference features—represented inthe space model—in the field of view of the optical sensor; and predictdeparture of the mobile device from the particular space accordingly.The mobile device can then: discard the space model for the particularspace; aggregate a new list of wireless APs now wirelessly accessible tothe mobile device; and send this new list of wireless APs and a requestfor location services to the remote computer system. The remote computersystem can then execute methods and techniques described above toestimate a new location of the mobile device and to return a secondspace model—for the new space now occupied by the mobile device—to themobile device.

Alternatively, in the foregoing implementation, the mobile device can:cache the (earlier) space model; download the second space modelresponsive to moving into a different space in the building; localizethe mobile device in the building based on both of these space models asthe user moves between these spaces; and only discard one of thesecached space models if the mobile device fails to determine that it hasentered the space represented by the model in more than a thresholdduration of time (e.g., four hours, one day).

11 PROTOCOL REPORTS

In one variation as shown in FIGS. 3 and 5 , the mobile device (or theremote computer system) compiles data collected during the instance ofthe protocol into a protocol report. The remote computer system can thenload this protocol report into a database or other repository to enablepost-hoc access, review, and verification of steps performed and datacollected during this instance of the protocol.

In one implementation, the mobile device: initializes a new protocolreport for the current instance of the protocol; writes identifiers ofthe user, the protocol, and the equipment unit to the new protocolreport; regularly tracks it location and orientation in the space(and/or relative to the equipment unit) during this instance of theprotocol; generates a sequence of timestamped geotags defining theselocations and orientations, such as at a rate of one geotag per secondduring the protocol; and writes this sequence of geotags to the newprotocol report, such as in real-time. The mobile device also writestimestamped step events to the new protocol report, includingidentifiers (e.g., step numbers) of each completed step, and the timesthat each completed step was opened, paused, and/or closed or otherwiseindicated complete by the user. Similarly, the mobile device can writetimestamped data capture events to the new protocol report, includingmanually-entered data values, automatically-derived data values, audioclips, images, and/or video clips recorded during each completed datacapture step of the protocol.

Upon completion of this instance of the protocol, the mobile device canupload this new protocol report to the remote computer system or to aremote database for storage. Alternatively, the mobile device can streamthese locations, step events, and data capture events to the remotecomputer system in (near) real-time during this instance of theprotocol, and the remote computer system can compile and store thesedata in a new protocol report for this instance of the protocol.

However, the mobile device and/or the remote computer system canimplement any other method or technique to record positions of themobile device, step events, and data capture events and to compile thesedata into a protocol record of the this instance of the protocol.

12. PROTOCOL REVIEW

A remote reviewer or verifier may then elect to review this instance ofthe protocol by replaying this protocol record. For example, a reviewerportal executing on the remote reviewer's computing device (e.g., adesktop computer) can: load a local copy of the protocol record orstream data from the protocol record from the remote database whenselected by the reviewer; render a 2D or 3D representation of the spacein which this instance of the protocol was performed; and render 2Dfootsteps or a 3D visualization of a human body in the 2D or 3Drepresentation of the space, respectively, according to locations andorientations defined in geotags in the protocol record. In particular,when the reviewer elects to “replay” the protocol record within thereviewer portal, the reviewer portal can render the 2D footstep or 3Dvisualization of a human body within the virtual representation of thespace in order of and timed according to timestamped geotags—definingthe mobile device's location and orientation during this instance of theprotocol—stored in the protocol report. During replay of the protocolrecord, the reviewer portal can also: identify a step of the protocolopen at the mobile device when the mobile device occupies the indicatedlocation in the virtual representation of the space, such as based ontimestamped event data contained in the protocol record; retrieve aprompt, guidance, or other specification for the current step; andrender this information over or adjacent the virtual representation ofthe space. Similarly, the reviewer portal can: identify an instance ofdata capture during this instance of the protocol based on timestampeddata capture events contained in the protocol record; extract capturedata from the protocol record for the current step depicted in thereviewer portal; and present this information to the reviewer, such asby rendering numerical data over a minimal dwell duration of 12 seconds,rendering textual data and static images over a minimal dwell durationof 30 seconds, and automatically replaying audio and video clipstime-synchronized to location and orientation playback over the virtualrepresentation of the space.

In the foregoing example, the reviewer portal can also replay theprotocol record at fixed or variable increased speed (e.g., at 2×, 4×,10×, or 60× of real-time), thereby enabling the reviewer to review andverify the instance of the protocol in less time than the protocolinstance itself. Alternatively, the reviewer portal can implement theforegoing methods and techniques to replay select segments of theprotocol record for the reviewer—such as around key events includingcompletion of each step and recordation of data during this instance ofthe protocol—thereby enabling the reviewer to review critical segmentsof the protocol record more likely to depict errors and thus reducetotal review time for this instance of the protocol.

Furthermore, throughout playback of the protocol record, the reviewerportal can record error flags entered by the reviewer, generatetimestamped error events synchronized to locations, step events, anddata capture events in the protocol record; and write these error eventsto the protocol record accordingly.

Therefore, the mobile device can generate a protocol record thatincludes relevant time, location, step, and data capture informationpackaged in a relatively small, robust file that: consumes limited localor remote storage (e.g., relative to a video of the whole instance ofthe protocol); can be accessed quickly for remote review andverification; and that still enables a reviewer or remote verifier toconfirm that the instance of the protocol was completed properly orotherwise identify and flag errors during this instance of the protocol.

Additionally or alternatively, the mobile device can stream data fromthe protocol record for the current instance of the protocol to thereviewer portal in real-time, and the reviewer portal can implement theforegoing methods and techniques to replay an animation of the currentinstance of the protocol—currently performed by the user—in (near)real-time, thereby enabling the reviewer to monitor the user's progress,manually verify or flag a step of the protocol for the user, andselectively enable the user to access subsequent steps of the protocol,such as while working remotely from the user.

12.1 Error Surfacing

In one variation shown in FIGS. 4 and 5 , the remote computer system:compares the protocol record for this instance of the protocol withprotocol records from previous instances of the protocol to identifydeviations between the current and previous instances of the protocol inBlock S150; and then selectively surfaces these deviations to thereviewer for verification in Block S152, such as by selectivelyreplaying segments of the protocol record spanning such deviations. Inparticular, the remote computer system can detect deviations between thecurrent and past instances of the protocol in various domains, such as:values of data captured; locations of the mobile device in the space orrelative to the equipment unit when data was captured; start times ofsteps in the protocol; duration of steps in the protocol; order oflocations occupied in the space or relative to the equipment unit;proximity to reference locations in the space or on the equipment unitlinked to steps in the protocol; spatial variance within steps in theprocess; and/or temporal variance within steps in the process; etc.Thus, when the remote computer system identifies a deviation between thecurrent and previous instances of the protocol based on comparisons ofprotocol records for these protocol instances, the remote computersystem can: selectively notify the reviewer of this deviation; andinterface with the reviewer portal to replay a segment of the protocolrecord for the current instance of the protocol that spans the stepcontaining this deviation.

In one implementation, the remote computer system: aggregates protocolrecords for past instances of the protocol performed at the sameequipment unit and/or at similar equipment units in the same space, inthe same building, on the same site, or at other sites affiliated withthe same entity; retrieves errors flags or verification confirmation foreach of these past instances of the protocol; generates one vector perprotocol record representing time, location, and data metrics from thisprotocol record; and labels each vector with its error flags orverification confirmation. The remote computer system then implementsregression, clustering, deep learning, and/or artificial intelligencetechniques to group like vectors and derive correlations between time,location, and data metrics and error and verification outcomes for thesepast instances of the protocol. Upon receipt of the protocol record forthe new instance of the protocol completed by the user, the remotecomputer system can: generate a similar vector for this new instance ofthe protocol and predict errors in this new instance of the protocolbased on correlations learned from the previous instances of theprotocol. For example, the remote computer system can: implementclustering techniques to identify a cluster of vectors—representingprevious instances of the protocol—nearest the new vector representingthe new instance of the protocol; and transfer error labels from thiscluster of vectors to the new vector. The remote computer system canthen: isolate a set of segments of the protocol record that (likely)depict these errors; serve a notification to the reviewer to review thisprotocol record; and serve or stream this set of segments of theprotocol record to the reviewer protocol for local replay for thereviewer.

In another implementation, the remote computer system: aggregatesprotocol records for past instances of the protocol performed at theequipment unit, etc.; filters this set of past protocol records toinclude only those verified and labeled as absent of errors; andimplements regression, deep learning, artificial intelligence, and/orother techniques to compress this filtered set of protocol records intoone reference protocol record, such as containing a range (or “band”) ofspatial, temporal, and data capture metrics that occur in verifiedinstances of the protocol (and that do not occur in erroneous steps inthese instances of the protocol). Upon receipt of a new protocol recordfor the new instance of the protocol completed by the user at the mobiledevice, the remote computer system can: compare spatial, temporal, anddata capture values stored in the new protocol record to the referencemobile device; identify values in the new protocol record that falloutside of the ranges defined in the reference protocol record; and flagthese values for review. The remote computer system can then interfacewith the reviewer portal to selectively serve segments of the newprotocol record in which these flagged values were recorded, such as: acomplete step in which a flagged value was recorded; or a one-minutesegment of the protocol record centered around a time that a flaggedvalue was recorded during this instance of the protocol.

Therefore, in the foregoing implementations, the remote computer systemcan: access a set of stored protocol records for a set of previousinstances of the protocol; detect a deviation between orders ofpositions of mobile devices represented in the first protocol record andrepresented in the set of stored protocol records; and then queue aparticular segment of the protocol record containing the deviation forverification by the reviewer or remote verifier. Similarly, the remotecomputer system can: access a reference protocol record for theprotocol; detect a deviation between position of the mobile deviceduring a particular step in the new instance of the protocol and areference location for the particular step defined in the referenceprotocol record; and then queue a segment of the protocol recordcontaining the deviation for remote verification if this deviationexceeds a threshold deviation (e.g., a threshold distance of one meter).In this example, the remote computer system can also: detect a deviationacross durations between confirmations of a first subset of steps in thenew instance of the protocol and confirmations of this same subset ofsteps defined in the reference protocol record; and queue a segment ofthe protocol record containing this deviation for remote verification ifthis deviation exceeds a threshold deviation (e.g., one minute). Morespecifically, in this example, the remote computer system can flag astep in the new instance of the protocol if the new protocol recordindicates that this step was completed in significantly more time orsignificantly less time than an average time or time range allocated forthis step in the reference protocol record.

Accordingly, the remote computer system can prompt the reviewer tomanually review these specific segments of the new instance of theprotocol represented in the new protocol record—such as rather than thenew protocol record in its entirety—and to identify errors or verifythese specific segments of the new instance of the protocol. For each ofthese segments, the reviewer portal can then: render an animation ofpositions of the mobile device—represented in the corresponding segmentof the new protocol record—over a spatial representation of theparticular space; render data—captured during a period of the firstinstance of the protocol record depicted in the first segment of theprotocol record—synchronized to the animation of positions of the mobiledevice; and serve a prompt to the remote verifier to verify accuratecompletion of one or more steps in the protocol depicted in thisanimated segment of the protocol record.

Upon receipt of error flags or verification of proper completion ofthese segments of the new instance of the protocol, the remote computersystem can label the new protocol record accordingly and then retraincorrelations between metrics contained in protocol records and erroroutcomes based on this new labeled protocol record. Additionally oralternatively, the remote computer system can recalculate ranges ofspatial, temporal, and data capture metrics that occur in verifiedinstances of the protocol (and that do not occur in erroneous steps inthese instances of the protocol) based on this new labeled protocolrecord.

However, the remote computer system can implement any other method ortechnique to: detect deviations between the new and past instances ofthe protocol or otherwise isolate segments of the new instance of theprotocol that may contain errors; and then selectively prompt a revieweror remote verify to review data in the new protocol record thatrepresented these targeted segments of the new instance of the protocol.

12.2 Protocol Record Comparison

In another variation, the reviewer portal accesses: a set of protocolrecords (e.g., a reference protocol record and new protocol record) forthe same protocol and for the same or similar equipment unit;synchronizes a start point of these protocol records (e.g., based ontimestamps of activation of a first step in these instances of theprotocol); and then implements the foregoing methods and techniques toreplay the set of protocol records concurrently within the same virtualrepresentation of the space (or the same virtual representation of avolume around the equipment unit). The reviewer portal can thus enablethe reviewer to visually compare the corresponding instances of theprotocols and quickly detect differences—and possible sources oferror—between these instances of the protocol.

The systems and methods described herein can be embodied and/orimplemented at least in part as a machine configured to receive acomputer-readable medium storing computer-readable instructions. Theinstructions can be executed by computer-executable componentsintegrated with the application, applet, host, server, network, website,communication service, communication interface,hardware/firmware/software elements of a operator computer or mobiledevice, wristband, smartphone, or any suitable combination thereof.Other systems and methods of the embodiment can be embodied and/orimplemented at least in part as a machine configured to receive acomputer-readable medium storing computer-readable instructions. Theinstructions can be executed by computer-executable componentsintegrated by computer-executable components integrated with apparatusesand networks of the type described above. The computer-readable mediumcan be stored on any suitable computer readable media such as RAMs,ROMs, flash memory, EEPROMs, optical devices (CD or DVD), hard drives,floppy drives, or any suitable device. The computer-executable componentcan be a processor but any suitable dedicated hardware device can(alternatively or additionally) execute the instructions.

As a person skilled in the art will recognize from the previous detaileddescription and from the figures and claims, modifications and changescan be made to the embodiments of the invention without departing fromthe scope of this invention as defined in the following claims.

I claim:
 1. A method for assisting execution of manual protocols atproduction equipment comprising: accessing a geospatial location of amobile device; identifying a particular site occupied by the mobiledevice based on the geospatial location; accessing identifiers of afirst set of wireless access points wirelessly accessible to the mobiledevice; based on identifiers of the set of wireless access points andknown locations of wireless access points within the particularbuilding, identifying a particular space within the particular buildingwithin the particular site occupied by the mobile device; loading aprotocol associated with an equipment unit in the particular spaceisolated and disconnected from local computer networks, the protocoldefining a sequence of steps for operation of the equipment unit by auser; detecting a first set of optical features in a field of view of anoptical sensor integrated into the mobile device; calculating a firstposition of the mobile device within the particular space at a firstresolution in response to proximity of the mobile device to a low-riskzone defined in a space model of the particular space and associatedwith a first step in the protocol, and based on positions of the firstset of optical features relative to a constellation of referencefeatures represented in the space model of the particular space; inresponse to the first position of the mobile device falling within afirst threshold distance of a first reference location proximal theequipment unit associated with the first step of the protocol, serving afirst guidance for the first step to the user, the first guidancecomprising a prompt to capture data at a second reference locationproximal the equipment unit following execution of a preceding controlaction at the first reference location proximal the equipment unit;serving a prompt to cease motion for a period of time in response toproximity of the mobile device to a high-risk zone defined in the spacemodel and associated with a second step in the protocol; detecting asecond set of optical features in the field of view of the opticalsensor during the period of time; calculating a second position of themobile device in proximity to the high-risk zone within the particularspace at a second resolution greater than the first resolution based onpositions of the second set of optical features relative to theconstellation of reference features represented in the space model ofthe particular space; and in response to the second position of themobile device falling within a second threshold distance of the secondreference location proximal a particular display of the equipment unitassociated with the second step of the protocol: accessing a secondimage captured by the optical sensor within the second thresholddistance of the second reference location of the equipment unit duringthe period of time; isolating a region in the second image correspondingto the particular display of the equipment unit; extracting a firstvalue from the region in the second image corresponding to a particularvalue presented on the particular display; and storing the first valuein a current protocol record containing a set of values captured duringa current instance of the protocol; and at a remote computer system, inresponse to identifying a first deviation between the set of values ofthe current protocol record and a previous set of values of previousinstances of the protocol, replaying a first segment of the currentprotocol record for the current instance of the protocol that spans thesecond step containing the first deviation.